Fake News Detection Using Transformer and Ensemble Learning Models

On social media, misinformation can spread quickly, posing serious problems. Understanding the content and sensitive nature of fake news and misinformation is critical to prevent the damage caused by them. To this end, the characteristics of information must first be discerned. In this paper, we propose a transformer-based hybrid ensemble model to detect misinformation on the Internet. First, false and true news on Covid-19 were analyzed, and various text classification tasks were performed to understand their content. The results were utilized in the proposed hybrid ensemble learning model. Our analysis revealed promising results, establishing the capability of the proposed system to detect misinformation on social media. The final model exhibited an excellent F1 score (0.98) and accuracy (0.97). The AUC (Area Under The Curve) score was also high at 0.98, and the ROC (Receiver Operating Characteristics) curve revealed that the true-positive rate of the data was close to one in this model. Thus, the proposed hybrid model was demonstrated to be successful in recognizing false information online. © 2022 IEEE.

Fake News Encoder Classifier (FNEC) for Online Published News Related to COVID-19 Vaccines

In the past few years, social media and online news platforms have played an essential role in distributing news content rapidly. Consequently. verification of the authenticity of news has become a major challenge. During the COVID-19 outbreak, misinformation and fake news were major sources of confusion and insecurity among the general public. In the first quarter of the year 2020, around 800 people died due to fake news relevant to COVID-19. The major goal of this research was to discover the best learning model for achieving high accuracy and performance. A novel case study of the Fake News Classification using ELECTRA model, which achieved 85.11% accuracy score, is thus reported in this manuscript. In addition to that, a new novel dataset called COVAX-Reality containing COVID-19 vaccinerelated news has been contributed. Using the COVAX-Reality dataset, the performance of FNEC is compared to several traditional learning models i.e., Support Vector Machine (SVM), Naive Bayes (NB), Passive Aggressive Classifier (PAC), Long Short-Term Memory (LSTM), Bi-directional LSTM (Bi-LSTM) and Bi-directional Encoder Representations from Transformers (BERT). For the evaluation of FNEC, standard metrics (Precision, Recall, Accuracy, and F1-Score) were utilized.

Analyzing COVID-Related Social Discourse on Twitter using Emotion, Sentiment, Political Bias, Stance, Veracity and Conspiracy Theories

Online misinformation has become a major concern in recent years, and it has been further emphasized during the COVID-19 pandemic. Social media platforms, such as Twitter, can be serious vectors of misinformation online. In order to better understand the spread of these fake-news, lies, deceptions, and rumours, we analyze the correlations between the following textual features in tweets: emotion, sentiment, political bias, stance, veracity and conspiracy theories. We train several transformer-based classifiers from multiple datasets to detect these textual features and identify potential correlations using conditional distributions of the labels. Our results show that the online discourse regarding some topics, such as COVID-19 regulations or conspiracy theories, is highly controversial and reflects the actual U.S. political landscape. © 2023 ACM.

CCTwins: A Weakly-Supervised Transformer-based Crowd Counting Method with Adaptive Scene Consistency Attention

Recently, crowd counting has attracted significant attention, particularly in the context of the COVID-19 pandemic, due to its ability to automatically provide accurate crowd numbers in images. To address the challenges of location-level labeling, several transformer-based crowd counting methods have been proposed with only count-level supervision. However, these methods directly use the transformer as an encoder without considering the uneven crowd distribution. To address this issue, we propose CCTwins, a novel transformer-based crowd counting method with only count-level supervision. Specifically, we introduce an adaptive scene consistency attention mechanism to enhance the transformer-based model Twins-SVT-L for feature extraction in crowded scenes. Additionally, we design a multi-level weakly-supervised loss function that generates estimated crowd numbers in a coarse-to-fine manner, making it more appropriate for weakly-supervised settings. Moreover, intermediate features supervised by count-level labels are utilized to fuse multi-scale features. Experimental results on four public datasets demonstrate that our proposed method outperforms the state-of-the-art weakly-supervised methods, achieving up to a 16.6% improvement in MAE and up to a 13.8% improvement in RMSE across all evaluation settings. Moreover, the proposed CCTwins obtains competitive counting performance, even when compared to the state-of-the-art fully-supervised methods. IEEE

Detecting COVID-19 Vaccine Stance and Symptom Reporting from Tweets using Contextual Embeddings

Determining sentiments of the public with regard to COVID-19 vaccines is crucial for nations to efficiently carry out vaccination drives and spread awareness. Hence, it is a field requiring accurate analysis and captures the interest of many researchers. Microblogs from social media websites such as Twitter sometimes contain colloquial expressions or terminology difficult to interpret making the task a challenging one. In this paper, we propose a method for multi-label text classification for the track of”Information Retrieval from Microblogs during Disasters (IRMiDis)” presented by the”Forum of Information Retrieval Evaluation” in 2022, related to vaccine sentiment among the public and reporting of someone experiencing COVID-19 symptoms. The following methodologies have been utilised: (i) Word2Vec and (ii) BERT, which uses contextual embedding rather than the fixed embedding used by conventional natural language models. For Task 1, the overall F1 score and Accuracy are 0.503 and 0.529, respectively, placing us fourth among all the teams, while for Task 2, they are 0.740 and 0.790, placing us second among all the teams who submitted their work. Our code is openly accessible through GitHub. 1 © 2022 Copyright for this paper by its authors.

Towards Predicting Length of Stay and Identification of Cohort Risk Factors Using Self-Attention-Based Transformers and Association Mining: COVID-19 as a Phenotype.

Predicting length of stay (LoS) and understanding its underlying factors is essential to minimizing the risk of hospital-acquired conditions, improving financial, operational, and clinical outcomes, and better managing future pandemics. The purpose of this study was to forecast patients' LoS using a deep learning model and to analyze cohorts of risk factors reducing or prolonging LoS. We employed various preprocessing techniques, SMOTE-N to balance data, and a TabTransformer model to forecast LoS. Finally, the Apriori algorithm was applied to analyze cohorts of risk factors influencing hospital LoS. The TabTransformer outperformed the base machine learning models in terms of F1 score (0.92), precision (0.83), recall (0.93), and accuracy (0.73) for the discharged dataset and F1 score (0.84), precision (0.75), recall (0.98), and accuracy (0.77) for the deceased dataset. The association mining algorithm was able to identify significant risk factors/indicators belonging to laboratory, X-ray, and clinical data, such as elevated LDH and D-dimer levels, lymphocyte count, and comorbidities such as hypertension and diabetes. It also reveals what treatments have reduced the symptoms of COVID-19 patients, leading to a reduction in LoS, particularly when no vaccines or medication, such as Paxlovid, were available.

ATFormer: Advanced transformer for medical image segmentation

Combining transformers and convolutional neural networks is considered one of the most important directions for tackling medical image segmentation problems. To learn the long-range dependencies and local contexts, previous approaches embedded a convolutional layer into feedforward neural network inside the transformer block. However, a common issue is the instability during training since large differences in amplitude across layers by pre-layer normalization. Furthermore, multi-scale features were directly fused using the transformer from the encoder to decoder, which could disrupt valuable information for segmentation. To address these concerns, we propose Advanced TransFormer (ATFormer), a novel hybrid architecture that combines convolutional neural networks and transformers for medical image segmentation. First, the traditional transformer block has been refined into an Advanced Transformer Block, which adopts post-layer normalization to obtain mild activation values and employs the scaled cosine attention with shifted window for accurate spatial information. Second, the Progressive Guided Fusion module is introduced to make multi-scale features more discriminative while reducing the computational complexity. Experimental results on the ACDC, COVID-19 CT-Seg, and Tumor datasets demonstrate the significant advantage of ATFormer over existing methods that rely solely on convolutional neural networks, transformers, or their combination.

Momentum contrast transformer for COVID-19 diagnosis with knowledge distillation.

Intelligent diagnosis has been widely studied in diagnosing novel corona virus disease (COVID-19). Existing deep models typically do not make full use of the global features such as large areas of ground glass opacities, and the local features such as local bronchiolectasis from the COVID-19 chest CT images, leading to unsatisfying recognition accuracy. To address this challenge, this paper proposes a novel method to diagnose COVID-19 using momentum contrast and knowledge distillation, termed MCT-KD. Our method takes advantage of Vision Transformer to design a momentum contrastive learning task to effectively extract global features from COVID-19 chest CT images. Moreover, in transfer and fine-tuning process, we integrate the locality of convolution into Vision Transformer via special knowledge distillation. These strategies enable the final Vision Transformer simultaneously focuses on global and local features from COVID-19 chest CT images. In addition, momentum contrastive learning is self-supervised learning, solving the problem that Vision Transformer is challenging to train on small datasets. Extensive experiments confirm the effectiveness of the proposed MCT-KD. In particular, our MCT-KD is able to achieve 87.43% and 96.94% accuracy on two publicly available datasets, respectively.

An amalgamation of vision transformer with convolutional neural network for automatic lung tumor segmentation.

Lung cancer has the highest mortality rate. Its diagnosis and treatment analysis depends upon the accurate segmentation of the tumor. It becomes tedious if done manually as radiologists are overburdened with numerous medical imaging tests due to the increase in cancer patients and the COVID pandemic. Automatic segmentation techniques play an essential role in assisting medical experts. The segmentation approaches based on convolutional neural networks have provided state-of-the-art performances. However, they cannot capture long-range relations due to the region-based convolutional operator. Vision Transformers can resolve this issue by capturing global multi-contextual features. To explore this advantageous feature of the vision transformer, we propose an approach for lung tumor segmentation using an amalgamation of the vision transformer and convolutional neural network. We design the network as an encoder-decoder structure with convolution blocks deployed in the initial layers of the encoder to capture the features carrying essential information and the corresponding blocks in the final layers of the decoder. The deeper layers utilize the transformer blocks with a self-attention mechanism to capture more detailed global feature maps. We use a recently proposed unified loss function that combines cross-entropy and dice-based losses for network optimization. We trained our network on a publicly available NSCLC-Radiomics dataset and tested its generalizability on our dataset collected from a local hospital. We could achieve average dice coefficients of 0.7468 and 0.6847 and Hausdorff distances of 15.336 and 17.435 on public and local test data, respectively.

Analysis of Learners' Sentiments on MOOC Forums using Natural Language Processing Techniques

MOOCs have gained a lot of popularity for past few years. Especially after the outbreak of Coronavirus, everyone is trying to gain some knowledge and skill while being at the comfort of home and making themselves safe. Due to sudden increase in the number of participants on MOOCs there is a need for an automated system to be able to assess the reviews and feedbacks given by the learners and find the sentiments behind their statements. This analysis will help trainers identify their shortcoming and make their courses even better. For sentiments analysis, several approaches may be used. This research aims to provide a system which will perform sentiments analysis on the novel dataset and show the comparison of lexicon-based vs transformer-based sentiment analysis models. For lexicon based, VADER was chosen and for transformer-based, state-of-The-Art BERT was chosen. BERT was found to be exceptionally good with an accuracy of 84% and F1-score of 0.64. © 2022 IEEE.

Transformer-based Deep Learning for COVID-19 Prediction Based on Climate Variables in Indonesia

Recent research on the effect of climate variables on coronavirus (COVID-19) transmission has emerged. Climate change can potentially cause new viral outbreaks, illness, and death. This study contributes to COVID-19 disease prevention efforts. This study makes two contributions: (1) we investigated the impact of climate variables on the number of COVID-19 cases in 34 Indonesian provinces, and (2) we developed a transformer-based deep learning model for time series forecasting for the number of positive COVID-19 cases the following day based on climate variables in 34 Indonesian provinces. We obtained data from March 15, 2020, to July 22, 2021, on the number of positive COVID-19 cases and climate change variables (wind, temperature, humidity) in Indonesia. To examine the effect of climate change on the number of positive COVID-19 cases, we employed 15 scenarios for training. The experiment results of the proposed model show that the combination of wind speed and humidity has a weakly positive correlation with positive COVID-19 incidence;however, the temperature has a considerably negative association with positive COVID-19 incidences. Compared to the other testing scenarios, the transformer-based deep learning model produced the lowest MAE of 175.96 and the lowest RMSE of 375.81. This study demonstrates that the transformer model works well in several provinces, such as Sumatra, Java, Papua, Bali, West Nusa Tenggara, East Nusa Tenggara, East Kalimantan, and Sulawesi, but not in Central Kalimantan, West Sulawesi, South Sulawesi, and North Sulawesi. © IJASEIT is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.

OutCLIP, A New Multi-Outfit CLIP Based Triplet Network

Choosing a proper outfit is one of the problems we deal with every day. Today, people tend to use online websites for shopping, and the COVID-19 situation forced this condition more than before. In this research, we proposed a new architecture for multi-fashion item retrieval from a website database. We deployed a CLIP transformer model instead of convolutional neural networks in a triplet network. We also added a long short-term memory network (LSTM) to automatically extract and code the image features to generate descriptive text for each input image. Our OutCLIP model succeeded in doing its task with 83% precision and 85% recall accuracy in multi-item retrieval. This model can be trained and used in fashion retrieval problems and improve the former proposed models. Considering the descriptive text and the image together gives the model a better understanding of the concept and improves its generalization. © 2023 IEEE.

Context Aware Automatic Subjective and Objective Question Generation using Fast Text to Text Transfer Learning

Online learning has gained a tremendous popularity in the last decade due to the facility to learn anytime, anything, anywhere from the ocean of web resources available. Especially the lockdown all over the world due to the Covid-19 pandemic has brought an enormous attention towards the online learning for value addition and skills development not only for the school/college students, but also to the working professionals. This massive growth in online learning has made the task of assessment very tedious and demands training, experience and resources. Automatic Question generation (AQG) techniques have been introduced to resolve this problem by deriving a question bank from the text documents. However, the performance of conventional AQG techniques is subject to the availability of large labelled training dataset. The requirement of deep linguistic knowledge for the generation of heuristic and hand-crafted rules to transform declarative sentence into interrogative sentence makes the problem further complicated. This paper presents a transfer learning-based text to text transformation model to generate the subjective and objective questions automatically from the text document. The proposed AQG model utilizes the Text-to-Text-Transfer-Transformer (T5) which reframes natural language processing tasks into a unified text-to-text-format and augments it with word sense disambiguation (WSD), ConceptNet and domain adaptation framework to improve the meaningfulness of the questions. Fast T5 library with beam-search decoding algorithm has been used here to reduce the model size and increase the speed of the model through quantization of the whole model by Open Neural Network Exchange (ONNX) framework. The keywords extraction in the proposed framework is performed using the Multipartite graphs to enhance the context awareness. The qualitative and quantitative performance of the proposed AQG model is evaluated through a comprehensive experimental analysis over the publicly available Squad dataset. © 2023, International Journal of Advanced Computer Science and Applications. All Rights Reserved.

Towards COVID-19 fake news detection using transformer-based models.

The COVID-19 pandemic has resulted in a surge of fake news, creating public health risks. However, developing an effective way to detect such news is challenging, especially when published news involves mixing true and false information. Detecting COVID-19 fake news has become a critical task in the field of natural language processing (NLP). This paper explores the effectiveness of several machine learning algorithms and fine-tuning pre-trained transformer-based models, including Bidirectional Encoder Representations from Transformers (BERT) and COVID-Twitter-BERT (CT-BERT), for COVID-19 fake news detection. We evaluate the performance of different downstream neural network structures, such as CNN and BiGRU layers, added on top of BERT and CT-BERT with frozen or unfrozen parameters. Our experiments on a real-world COVID-19 fake news dataset demonstrate that incorporating BiGRU on top of the CT-BERT model achieves outstanding performance, with a state-of-the-art F1 score of 98%. These results have significant implications for mitigating the spread of COVID-19 misinformation and highlight the potential of advanced machine learning models for fake news detection.

Design and Development of Toroidal Core Winding Machine with Digital Interface

The small size and inherent superior electrical characteristics of a toroid has made it the first choice for many Original Equipment Manufacturers (OEMs). However, the lack of knowledge regarding the toroidal coil winding equipment is still hampering the growth of toroid as the first choice for transformers, inductors and other electrical applications. Additionally, due to Covid-19 pandemic and lockdown situation, small scale companies are lacking skilled manpower for the high precision task of toroidal core winding and taping. Although the machine is readily available in the market, the cost is still very high. Toroidal core winding machine is an equipment used for the purpose of winding toroidal cores which is used in various electrical machines such as current transformers, power transformers, isolation transformers, inductors and chokes, auto transformers, etc. This project aims to develop a low-cost toroidal winding machine with a user-friendly digital interface for selection of winding parameters as per the user input. The winding machine developed in this project is efficient and reliable with high-speed performance and negligible error. © 2022 IEEE.

Ensemble Learning using Transformers and Convolutional Networks for Masked Face Recognition

Wearing a face mask is one of the adjustments we had to follow to reduce the spread of the coronavirus. Having our faces covered by masks constantly has driven the need to understand and investigate how this behavior affects the recognition capability of face recognition systems. Current face recognition systems have extremely high accuracy when dealing with unconstrained general face recognition cases but do not generalize well with occluded masked faces. In this work, we propose a system for masked face recognition. The proposed system comprises two Convolutional Neural Network (CNN) models and two Transformer models. The CNN models have been fine-tuned on FaceNet pre-trained model. We ensemble the predictions of the four models using the majority voting technique to identify the person with the mask. The proposed system has been evaluated on a synthetically masked LFW dataset created in this work. The best accuracy is obtained using the ensembled models with an accuracy of 92%. This recognition rate outperformed the accuracy of other models and it shows the correctness and robustness of the proposed model for recognizing masked faces. The code and data are available at https://github.com/Hamzah-Luqman/MFR. © 2022 IEEE.

Estimating time-series changes in social sentiment @Twitter in U.S. metropolises during the COVID-19 pandemic.

Since early 2020, the global coronavirus pandemic has strained economic activities and traditional lifestyles. For such emergencies, our paper proposes a social sentiment estimation model that changes in response to infection conditions and state government orders. By designing mediation keywords that do not directly evoke coronavirus, it is possible to observe sentiment waveforms that vary as confirmed cases increase or decrease and as behavioral restrictions are ordered or lifted over a long period. The model demonstrates guaranteed performance with transformer-based neural network models and has been validated in New York City, Los Angeles, and Chicago, given that coronavirus infections explode in overcrowded cities. The time-series of the extracted social sentiment reflected the infection conditions of each city during the 2-year period from pre-pandemic to the new normal and shows a concurrency of waveforms common to the three cities. The methods of this paper could be applied not only to analysis of the COVID-19 pandemic but also to analyses of a wide range of emergencies and they could be a policy support tool that complements traditional surveys in the future.

Attention-based Clinical Note Summarization

In recent years, the trend of deploying digital systems in numerous industries has hiked. The health sector has observed an extensive adoption of digital systems and services that generate significant medical records. Electronic health records contain valuable information for prospective and retrospective analysis that is often not entirely exploited because of the complicated dense information storage. The crude purpose of condensing health records is to select the information that holds most characteristics of the original documents based on a reported disease. These summaries may boost diagnosis and save a doctor's time during a saturated workload situation like the COVID-19 pandemic. In this paper, we are applying a multi-head attention-based mechanism to perform extractive summarization of meaningful phrases on clinical notes. Our method finds major sentences for a summary by correlating tokens, segments, and positional embeddings of sentences in a clinical note. The model outputs attention scores that are statistically transformed to extract critical phrases for visualization on the heat-mapping tool and for human use.

Enhancing Multi-disease Diagnosis of Chest X-rays with Advanced Deep-learning Networks in Real-world Data.

The current artificial intelligence (AI) models are still insufficient in multi-disease diagnosis for real-world data, which always present a long-tail distribution. To tackle this issue, a long-tail public dataset, "ChestX-ray14," which involved fourteen (14) disease labels, was randomly divided into the train, validation, and test sets with ratios of 0.7, 0.1, and 0.2. Two pretrained state-of-the-art networks, EfficientNet-b5 and CoAtNet-0-rw, were chosen as the backbones. After the fully-connected layer, a final layer of 14 sigmoid activation units was added to output each disease's diagnosis. To achieve better adaptive learning, a novel loss (Lours) was designed, which coalesced reweighting and tail sample focus. For comparison, a pretrained ResNet50 network with weighted binary cross-entropy loss (LWBCE) was used as a baseline, which showed the best performance in a previous study. The overall and individual areas under the receiver operating curve (AUROC) for each disease label were evaluated and compared among different models. Group-score-weighted class activation mapping (Group-CAM) is applied for visual interpretations. As a result, the pretrained CoAtNet-0-rw + Lours showed the best overall AUROC of 0.842, significantly higher than ResNet50 + LWBCE (AUROC: 0.811, p = 0.037). Group-CAM presented that the model could pay the proper attention to lesions for most disease labels (e.g., atelectasis, edema, effusion) but wrong attention for the other labels, such as pneumothorax; meanwhile, mislabeling of the dataset was found. Overall, this study presented an advanced AI diagnostic model achieving a significant improvement in the multi-disease diagnosis of chest X-rays, particularly in real-world data with challenging long-tail distributions.

Aspect-Based Sentiment Analysis with Semi-Supervised Approach on Taiwan Social Distancing App User Reviews

Sentiment analysis has a critical role to reveal an opinion in a text-based form. Therefore, we exploit this analysis to discover the sentiment polarity of Taiwan Social Distancing mobile application. This paper proposes a semi-supervised scheme for annotating this mobile application's reviews. The semi-supervised scheme utilized a combination of numeric rating and lexicon-based sentiment. In addition, we also perform the sentiment analysis on an aspect-based level. Based on the experiment, we decide to select three aspects to be analyzed. This paper also evaluates the proposed scheme by implementing bidirectional encoder representations from transformers (BERT) and multilayer perceptron (MLP) as the classification model using the sentiment label of the proposed scheme. The result shows that the annotation of the proposed scheme outperforms the data annotation using counterpart models. © 2023 IEEE.